Symmetrization and Rademacher Complexity

Let’s first see how primitive covers were inadequate. Recall that a function class G is a primitive cover for a function class F at scale > 0 over some set S if: • G ⊆ F , • |G| <∞, and • for every f ∈ F there exists g ∈ G with supx∈S |g(x)− f(x)| ≤ . Last class, we gave a generalization bound for classes with primitive covers (basically, primitive covers give discretizations, and then we apply finite class generalization). Problems with primitive covers. It’s pretty easy to run into limits of this technique. • Consider linear predictors as before, but the points x ∈ R are from some unbounded distribution, for instance a Gaussian. This immediately breaks the earlier construction. One fix is to truncate the distribution: since Gaussians concentrate well, we can find an X so that ‖x‖2 ≤ X with probability at least 1− δ (and this X does not depend too badly on n: recall from homework 1 the analysis of the maximum of a collection of scalar Gaussian random variables). So now we can first condition away an event of probability at most δ that some points have ‖x‖2 > X, and then run the cover argument as before. • Consider discontinuous function classes, for instance w 7→ sgn(〈w, x〉). If < 2, for any linear classifier f there must exist gf that exactly agrees with f on every point (i.e., any < 2 may as well be = 0). Since for any x 6 = 0 and w 6= 0, sgn(〈w, x〉) 6= sgn(〈−w, x〉), it follows that the primitive covering number is again infinite (e.g., for any w 6= 0, the only vectors within < 2 for this metric is the set {cw : c ∈ R \ {0}}, so the cover must include one vector for each direction, as well as 0). There are a number of ways to fix this (including giving non-primitive covers); we will come back to it after discussing Rademacher complexity. There is a better notion of cover that fixes these, but we’ll get there through Rademacher complexity.